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ABSTRACT: Four years of data on the 
academic performance of 256 students who 
self-selected online developmental writing 
rather than a face-to-face section (about 10% 
of the 2,275 students enrolled in the course 
overall) are examined in this empirical 
study. The research controls for self-selec- 
tion effects related to demographic vari- 
ables, student status, and academic pre- 
paredness. Resulting analysis of the data 
suggests that instructional delivery 
method — asynchronous or face-to-face — has 
a significant impact on student outcomes. 
The researchers summarize findings related 
to the influences of various factors on the 
retention and success of students in the 
online course. Based on these findings, the 
researchers offer suggestions for improving 
or creating a new online developmental 
course and discuss implications for future 
research. 

The rapid growth of online education 
across the nation is producing a challenge for 
many developmental education programs. 
For various reasons, perhaps primarily due 
to a reticence to create more hurdles for a 
population already facing many, developmen- 
tal writing faculty and administrators are of- 
ten reluctant to offer online (asynchronous) 
versions of traditional face-to-face develop- 
mental writing courses. Recognizing that re- 
luctance, we do not know whether it is truly 
justified. 

Two main concerns faculty and adminis- 
tration often express about creating online 
versions of developmental courses are fears 
about retention and students’ ability to handle 
the technology. However, a study by Jennings 
(2002) of the computer attitudes of develop- 
mental students concluded that developmen- 
tal students “reported generally more posi- 
tive attitudes toward computers than did 
nondeveiopmentai students” and speculated 
that this positive attitude “may enhance the 
probability for advancement into the main- 
stream college curricula” (p. 5). 

Saxon and Boylan (2003) emphasized the 
importance of “research on the impact of 
online and other distance learning method- 
ologies for developmental education” (p. 3). 
Bailey (2002) also noted that “Most colleges 


are in the early stages of offering web-based 
courses...” and “have not yet systematically 
investigated students’ reasons for dropping 
online courses” (p. 5). Morris (2001) re- 
searched factors that affect student retention 
in online writing courses, and emphasized that 
“in general literature (including education- 
specific literature) perceptions of attrition are 
fraught with vagueness and potentials for in- 
adequate understanding.” He stated further 
that “attrition in online classes frequently is 
reported to be higher than in matching face- 
to-face classes, but the data is mixed, derives 
from different and incompatible measuring 
methods, is contradictory, and suggests re- 
search difficulty because of multiple variables” 
(p. 5). 

In support of online instruction, the 
Michigan Community College Association 
(2004) — in cooperation with the Michigan Vir- 
tual University (MVU) — has created a virtual 
learning collaborative among the state’s 28 
community colleges, listing online courses and 
allowing students to take them at any Michi- 
gan community college while maintaining 
academic record and residency status at the 
home college. A Fall 2003 review of the online 
courses offered through MVU and subsequent 
visits to the college websites have indicated 
that, although 19 colleges (70% of those of- 
fering online writing) offered freshman writ- 
ing online, only one listed developmental 
writing online. 

Before more colleges will be interested 
in offering developmental writing online, 
many instructors and administrators need to 
know whether it is as productive of desired 
student outcomes as traditional, face-to-face 
instruction. The researchers conducted the 
following study to help address these con- 
cerns. 

Methodology 

Developmental Writing Placement 

This study involved an urban, Midwest- 
ern community college with 9,000 full-time- 
equivalent students. Several years ago, the 
college established basic skill “levels” as pre- 
requisites for taking courses that lead to a 
degree or transfer. For both reading and writ- 
ing, skill levels are established primarily 
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through reviewing student scores on the Col- 
lege Board’s ACCUPLACER™ Computerized 
Placement Tests (CRT; College Board for Edu- 
cators, 2004); students with extremely low 
English language skills are tested with the 
Levels of English Proficiency (College Board for 
Educators, 2004). Students whose scores fall 
below the required cut-score range raise their 
skill levels primarily through taking develop- 
mental courses with “exit competencies." Cur- 
rently, a CRT writing score of 65-85 (and a 
reading score of 59-77) would place a student 
in Lansing Community College’s developmen- 
tal Writing 117, and a student completing the 
course with a grade of 2.5 or higher (on a 4.0 
scale) would earn an automatic level 6 (i.e., 
“college-ready” In writing), enabling the stu- 
dent to take freshman composition or any 
other course with a level-6 writing prerequi- 
site. Though Writing 117 is taught by many 
faculty members, all sections use a common 
syllabus, text, and exit portfolio (which must 
be “passed” by two external readers). For the 
past 4 years the college has offered Writing 
117 asynchronously. Any student who is 
placed into Writing 117 can register for an 
online section of the course, and, over the 4 
years of this research (academic records from 
Fall 1999 through Summer 2003), 256 stu- 
dents chose that option, approximately 10% 
of the total number enrolled In the course. 

Participants 

The participant profile indicated that of 
the 256 students taking online Writing 117, 
33.3% were over age 25 (compared to 18.2% 
of those in face-to-face sections); 68.2% were 
female (compared to 54.2% In face-to-face 
sections); 33.3% were full time (compared to 
52.2% in face-to-face sections); 26.7% were 
minorities (compared to 41% in face-to-face 
sections); 56.1% already had a college-level 
reading score, that is, 78 or higher on the 
ACCUPLACER™ Reading Test (College Board 
for Educators, 2004); compared to 44% for 
face-to-face sections); 22.2% had a college-level 
writing score, that is, 86 or above on the 
ACCUPLACER™ Writing Test (College Board 
for Educators, 2004 (compared to 11.6% In 
face-to-face sections). In summary, the stu- 
dents who self-selected for online tended to 
be older, female, white, and part time, with 
higher reading and writing placement scores. 

Design and Procedure 

Although there is clearly a sense In which 
the student who withdraws from a course did 
not have a “successful” experience in the 
course, we have observed that many of the 
reasons students withdraw from courses are 
unrelated to their ability to master success- 
fully the academic challenges of the course. 


Given the goal of improving student success, 
there Is a clear need for researchers and prac- 
titioners to distinguish between students who 
withdraw and those who are retained but are 
unable to meet the academic demands of the 
course successfully. Educators cannot afford 
to assume that the forces influencing reten- 
tion are identical to those influencing success. 

Since we recognized differences between 
reasons for noncompletion, data on comple- 
tion status were used to construct two binary 
student outcome variables: retention to course 
completion (whether or not the student with- 
drew), and, for those students that were re- 
tained and received a grade, whether or not 
the student was successful, with “success” de- 
fined as receiving a grade of 2.5 or higher for 
the course. Students who received an “incom- 
plete” were excluded from the analysis since, 
although they were retained, there was no way 
to measure their level of success. 

Initial analysis showed that online and 
face-to-face sections differed significantly with 


The factors that influ- 
ence success are not iden- 
tical with the factors that 
influence withdrawal. 


respect to retention (p < 0.0001) and success 
(p < 0.0001) rates, with online sections hav- 
ing a greater withdrawal rate but also a higher 
success rate for those who completed the 
course. However, since developmental writing 
students were not randomly assigned to online 
versus face-to-face instructional delivery meth- 
ods, but rather self-selected a delivery method, 
a simple comparison of these two methods 
with respect to student outcomes was insuffi- 
cient. A simple comparison would not isolate 
the impact of delivery method due to the lack 
of control for potentially confounding vari- 
ables associated with self-selection. Analysis 
revealed that the online and face-to-face stu- 
dents also differed significantly with respect 
to demographic, student status, and prepared- 
ness variables: ethnicity (relatively more whites 
and fewer minorities in online, p < 0.0001), 
sex (relatively more females and fewer males 
In online, p < 0.0001), age (relatively more 
older and fewer younger students In online, p 

< 0.0001), credit load (relatively more part- 
time and fewer full-time students in online, p 

< 0.0001), and preparedness (relatively more 
students with high reading and writing CRT 
scores in online, p < 0.0001 for reading and p 

< 0.01 for writing). Late-registering students 
were often forced into online sections after 
face-to-face sections filled up; although this 
led to a higher proportion of late-registering 


students in online sections, the difference was 
not statistically significant. 

Given that the students In online sections 
differed significantly from students in face- 
to-face sections with respect to some demo- 
graphic, student status, and academic pre- 
paredness variables, there was a clear need to 
control for the influence of these variables In 
any analysis directed at identifying the impact 
of delivery method. Consequently, sex, 
ethnicity (coded as white or minority), age, 
load (full time or part time), time of registra- 
tion (early or late, with late defined as regis- 
tration the week classes began or later), and 
reading, writing, and math computerized 
placement test scores were included in the 
models as control variables in addition to the 
predictor of chief interest: delivery method 
(online or face-to-face). 

Since both student outcomes were binary 
variables, and the model included both dis- 
crete and interval-level variables, logistic re- 
gression was used to model the student out- 
comes. Odds ratios, the typical association 
measures in categorical data analysis, were 
used as effect size indicators because of their 
relatively simple and direct relationship to 
parameter estimates (Agresti, 1990, pp. 91- 
92). 

The data concerning the completion and 
success of the students in the course used to 
construct the outcome variables were clus- 
tered, since student outcomes occurred in par- 
ticular course sections. Observations within 
clusters are often more alike than observations 
between clusters, leading to correlation of 
observations within clusters. Hence, under- 
estimation of standard errors, with corre- 
sponding overestimation of significance lev- 
els, can result with clustered data (Angeles & 
Mroz, 2001, pp. 2, 5-8; Huber, 1967) since stan- 
dard estimation procedures assume Indepen- 
dence of observations. To address this, gen- 
eralized estimating equations were used to 
produce robust standard error estimates. This 
estimating procedure modifies the standard 
iteratively re-weighted least squares algorithm 
for generating maximum likelihood estimates 
by replacing the standard weight matrix with 
a weight matrix having nonzero off-diagonal 
elements that reflect the correlations among 
the observations at the cluster level. 

Since the models that were estimated In- 
cluded continuous as well as categorical vari- 
ables, the conditions for the standard good- 
ness-of-fit tests were not met because of in- 
sufficiently large expected frequencies for 
each combination of the explanatory vari- 
ables. We chose the Hosmer-Lemeshow 
“declles-of-rlsk” test as the preferred alterna- 
tive in this case, which compares model-pre- 
dicted frequencies with actual frequencies in 
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Table 1 

Effects (Success as Outcome) 

Effect 

Adjusted 
Odds Ratio 

950/0 

Confidence 

Interval 

Delivery method (online vs. face-to-face) 

4.17** 

1.37-12.50 

Sex (females vs. males) 

2.00** 

1.53-2.63 

Ethnicity (white vs. minority) 

1.64** 

1.25-2.13 

Time of registration (late vs. early) 

1.16 

0.79-1.72 

Credit load (full time vs. part time) 

1.12 

0.86-1.45 

Age 

1.02 

0.99-1.04 

Writing CPT score 

1.02** 

1.01-1.04 

Math CPT score 

1.01** 

1.003-1.018 

Reading CPT score 

1.01* 

1.001-1.025 

*p < 0.05 **p < 0.01 
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10 groups ordered by estimated probabilities 
on the outcome variable. In the case of the 
Hosmer-Lemeshow test (sometimes called a 
“lack-of-fit” test), a statistically insignificant 
result is desired, indicating insufficient evi- 
dence to conclude that the model does not 
fit the data adequately. 

Models with interaction terms involving 
delivery method and the control variables 
were fitted to the data to test for the presence 
of interaction effects involving delivery 
method, though no reliable evidence of in- 
teraction effects was found. Since no signifi- 
cant interaction effects were detected, the re- 
sults presented and interpreted were derived 
from main effects models. 

Finally, it should be noted that the mea- 
surement of the outcomes — retention and suc- 
cess — occurred after measurement of the pre- 
dictor variables. The nature of the predictor 
variables was such that it made it difficult to 
conceive of any causal mechanisms that would 
cause them to be subject to be influenced by 
the outcomes. Consequently, any association 
between variables in the model and the out- 
comes could not reflect any causal influence 
of the outcomes on values of the predictor 
variables, but rather demonstrated an influ- 
ence of student characteristics reflected in the 
predictors on the outcomes. 

Results 

Success as Outcome 

The model produced an acceptable fit 
with the data, with a p-value of 0.37 for the 
Hosmer-Lemeshow (HL) goodness-of-fit test. 
Table 2 contains the effect size measures (odds 
ratios) and 95% confidence intervals. The 
effect size, as reported in odds ratios, refers 
to the magnitude of associations; for example, 
the odds ratio of 2.0 reported for sex reveals 
that the sex of the student had a stronger as- 


sociation with suc- 
cess than did credit 
load, which pro- 
duced an odds ratio 
of 1.12 (see Table 
1). Delivery method 
remained a signifi- 
cant predictor of 
success even after 
adjusting for demo- 
graphic and student 
status variables and 
placement test 
scores, with online 
students being sig- 
nificantly more 
likely than face-to- 
face students to suc- 
ceed in the course (odds ratio 4.17, p<0.0001). 

Of the demographic variables indepen- 
dent of delivery method, only sex and 
ethnicity were significant predictors of suc- 
cess. The adjusted odds ratio of success for 
females was significantly higher than for 
males. In the case of ethnicity, the adjusted 
odds ratio of success for white students was 
significantly higher than for minority stu- 
dents. Concerning preparedness, OPT scores 
were shown to be significantly and positively 
related to success, with writing placement 
score the most important predictor. Some- 
what surprisingly, math placement test score 
was also significantly positively related to suc- 
cess. 

Retention as Outcome 

Results for the HL goodness-of-fit test (p 
= 0.21) indicated no significant lack-of-fit with 
the data. The effect size measures (odds ra- 
tios) and 95% confidence intervals for reten- 
tion as the outcome measure were also calcu- 
lated (see Table 2). 

After adjusting for demographic differ- 
ences, student credit load, time of registra- 
tion, and preparedness level, the delivery 
method remained 
a strong predictor 
of retention, with 
face-to-face stu- 
dents being signifi- 
cantly more likely 
than online stu- 
dents to finish the 
course (odds ratio 
4.76, p < 0.0001). 

Full-time students 
were significantly 
more likely than 
part-time students 
to be retained 
(odds ratio 1.96, p 
< 0.001), and fe- 


males were more likely to be retained than 
males (odds ratio 1.72, p < 0.01). 

Discussion 

A key result of this research is the signifi- 
cance of delivery method as a predictor of 
student outcomes after adjusting for any self- 
selection effects associated with the demo- 
graphic, student status, and preparedness vari- 
ables. Assuming that all important self-selec- 
tion effects have been controlled, this result 
suggests that there is something about the 
dynamics of online instruction, construed 
broadly (by comparison with traditional, face- 
to-face instruction), which has an impact on 
student outcomes. More specifically, it sug- 
gests that there are some things about online 
instruction as a delivery method that lead to 
greater withdrawal rates but that may ulti- 
mately lead to success rates for those students 
who finish the course that are comparable to, 
if not greater than, success rates in traditional, 
face-to-face instruction. 

Of particular value for researchers and 
practitioners is the finding of evidence sug- 
gesting that the factors that influence success 
are not identical with the factors that influ- 
ence withdrawal. Particularly noteworthy here 
is the significance of placement test scores as 
predictors of success (but not of withdrawal) 
and the significance of credit load as a pre- 
dictor of withdrawal (but not of success). 

Consideration should be given to the pos- 
sibility that the greater withdrawal rate in 
online sections may be indirectly causally re- 
lated to the greater success rate in those sec- 
tions. Students in online sections who com- 
pleted the course had a higher mean writing 
OPT score than students in face-to-face sec- 
tions who completed the course, 78.9 versus 
77.2. An analysis, stratifying on delivery 
method, found that, although students with 
lower writing OPT scores withdrew from 
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Table 2 

Effects (Retention as Outcome) 

Effect 

Adjusted 
Odds Ratio 

95% 

Confidence 

Interval 

Delivery method (face-to-face vs. online) 

4.76** 

2.70-8.33 

Credit load (full time vs. part time) 

1.96** 

1.30-3.03 

Sex (females vs. males) 

1.72** 

1.18-2.50 

Ethnicity (white vs. minority) 

1.33 

0.93-1.91 

Time of registration (early vs. late) 

1.14 

0.72-1.82 

Age 

1.02 

0.99-1.05 

Math CPT score 

1.01 

0.98-1.02 

Reading CPT score 

1.01 

0.99-1.02 

Writing CPT score 

0.99 

0.97-1.00 

**p < 0.01 
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online sections in greater proportions than 
students with higher writing CPT scores (odds 
ratio of withdrawal, low vs. high CPT scores = 
1.47, p = 0.34), exactly the reverse association 
existed in face-to-face sections: students with 
higher writing CPT scores withdrew from face- 
to-face sections in greater proportions than 
students with lower writing CPT scores (odds 
ratio = 1.35, p = 0.19), although the differ- 
ence did not reach statistical significance. 

A similar pattern of results was found for 
reading CPT score. Students in online sections 
who completed the course also had a higher 
mean reading CPT score than students in face- 
to-face sections who completed the course, 
81.3 versus 76.1. Employing the same strati- 
fied analysis that controls for delivery method, 
whereas students with low reading CPT scores 
withdrew at rates essentially similar to those 
with high reading CPT scores in face-to-face 
sections (odds ratio 0.99, p = 0.93), students 
with low reading CPT scores withdrew at 
higher rates than those with high reading CPT 
scores in online sections (odds ratio 1.48, p = 
0.18), although the difference did not reach 
statistical significance. 

Although neither pattern of results was 
statistically significant, the relatively low power 
of the tests for online students {N - 227 and 
256, respectively, for writing and reading CPT 
scores) raises the question whether 
these findings might be indicative of 
something in the dynamics of online 
instruction (by contrast with face-to- 
face instruction) that signals more 
clearly to students whether their level 
of preparedness is adequate to the 
demands of the course. 

Limitations 

An important limitation of this 
study flows from the fact that virtu- 
ally all (96%) of the students in the 
online sections were taught by the 
same instructor, who also taught fewer 
than 2% of the students in the face- 
to-face sample. Therefore, a strong 
association between this instructor 
and delivery method exists, which 
makes it difficult to unambiguously 
distinguish the effects of delivery 
method from the effects of this par- 
ticular instructor on student out- 
comes. The results of additional analy- 
sis, however, failed to provide good 
reasons for thinking that we have an 
instructor effect masquerading as a 
delivery method effect. 

Analysis of the subsample of face- 
to-face students revealed that, al- 


though this particular instructor’s face-to-face 
students showed higher rates of success than 
the face-to-face students of other instructors 
(72.7% versus 64.3%), the difference was not 
statistically significant. The associations be- 
tween delivery method and success for stu- 
dents of this instructor and for students of 
the other instructors were comparable in 
magnitude: odds ratios (online versus face- 
to-face students) of 3.78 for this instructor and 
3.33 for the students of the other instructors, 
a difference that was not statistically signifi- 
cant (p = 0.91). 

The same analysis with respect to reten- 
tion also failed to provide strong evidence of 
an instructor effect. The nature of the asso- 
ciation between delivery method and with- 
drawal status differed little between the stu- 
dents of this particular instructor and the stu- 
dents of the other instructors: odds ratio (of 
withdrawal, online versus face-to-face) of 4.40 
for this particular instructor and an odds ra- 
tio of 4.67 for students of other instructors. 
This difference was not statistically significant 
(p = 0.94). However, the very small number of 
online students taught by other instructors 
and the very small number of face-to-face stu- 
dents taught by this particular instructor re- 
sulted in tests of relatively low power. 

A relevant question concerns whether all 
the important variables related to self-selec- 
tion have been controlled. For example, stu- 


dent expectations are not controlled in the 
analysis. It is conceivable that students select- 
ing online instruction may be more likely than 
students selecting face-to-face instruction to 
have unreasonable expectations about the 
difficulty of such courses (e.g., are unduly in- 
fluenced by the fact that online sections are 
more accessible and convenient than face-to- 
face sections), and this may well lead to higher 
withdrawal rates in online courses after expo- 
sure to the level of difficulty of the 
coursework. Although there may be other 
variables associated with self-selection of de- 
livery method that are also associated with 
student outcomes, it is reasonable to conclude 
that delivery method may have at least some 
influence on student outcomes — particularly 
concerning retention, where the effect of de- 
livery method was particularly strong. The 
results suggest that there may be something 
different in the dynamics of online instruc- 
tion in developmental writing which leads to 
higher rates of withdrawal. 

Implications for Pedagogy 

The findings of this study suggest some 
implications for pedagogy. The following 
table highlights suggestions that might be 
helpful for improving an existing online de- 
velopmental writing course or for creating a 
new one. 
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Table 3 

Findings and Implications for Pedagogy 

Finding 

Implications 

Suggestions 

"Withdrawal” rate 
is higher in online 
than in face-to-face 
sections. 

Implement strategies to 
Increase retention. 

- require an online orientation 

- require students to complete tasks online at the start of 
the course (e.g., sending an e-mail, attaching a document) 

- make expectations for workload clear 

- make expectations for technology clear 

- be liberal with "incomplete” grades 

Reading level 
and writing level 
(CPT scores) 
affect success. 

Make written 
communication and 
assigned readings 
accessible. 

- test readability of instructor-generated text and 
course packs 

- provide study guides, tips, summaries 

- make course sites easy to navigate 


Continue to use the CRTs. 


Full-time students 
are more likely to 
complete the 
course. 

Implement strategies to 
Increase part-time 
student retention. 

- create a student questionnaire to study 
environmental distractions and their relationship likely 
to success 

- inform counseling staff that part-time students are 
more at-risk 

- form online support groups for part-time students 

- create an online "Question & Answer” document 

for access to college and community support services 

Students who 
complete online 
tend to be as 
successful as (or 
more successful than) 
students who complete 
tace-to-face. 

Continue offering 
online developmental 
writing. 

- present at conferences 

- create listserv for faculty teaching developmental 
writing online 

- publish outside the institution 
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Implications for Future 
Research 

■ Investigate current cut-score for college- 
level reading (ACCUPLACER, 78-120) in 
relation to success in developmental writ- 
ing. (Do we need to have students reach 
college level in reading before they take 
developmental writing?) 

■ Explore results of college reading level and 
success in online courses overall. (Is the 
current reading level adequate for online 
courses? Could it be set too low and con- 
tributing to course withdrawal?) 

■ Use this research as a base line and create 
a study for succeeding years. (Can we iden- 
tify trends?) 

Conclusions 

Our effort to adjust the study results for 
possible self-selection effects and to control 
for a possible instructor effect has resulted in 
a picture that is not substantially different 
from the initial unadjusted comparison of 
online and face-to-face developmental writing 
students. However, it did result in a weaker 
association between delivery method and re- 
tention and a stronger association between 
delivery method and success when compared 
with the unadjusted results. Although an as- 
pect of the data set (very limited variation in 
instructors in the online sample) limits our 
ability to be definitive, the results provide 
good reasons for thinking that, when it comes 
to student outcomes of retention and success, 
delivery method matters. Although online 
students who are retained are more likely to 
be successful than retained face-to-face stu- 
dents, the online, asynchronous delivery 
method appears less successful than the face- 
to-face method in retaining students to course 
completion. 

Although we found evidence that online 
and face-to-face sections differ in the produc- 
tion of student outcomes, the absence of sig- 
nificant interaction effects with delivery 
method leads us to conclude that at least some 
of the forces influencing student outcomes 
are the same in online and face-to-face devel- 
opmental writing. Identifying these common 
forces is important for efforts to increase the 
proportion of developmental writing students 
that have successful experiences. 

Perhaps equally important, our results 
also suggest that the dynamics producing stu- 
dent outcomes — retention to course comple- 
tion and success — are not identical. Improv- 
ing student success in developmental writing 
requires attention to factors that differ to some 
extent from those factors that appear impor- 


tant for improving student retention. We have 
found that, although placement test scores 
appear significantly and positively related to 
the odds of success, they are not significantly 
related to retention. On the other hand, al- 
though credit load has not been found to be 
significantly related to success, it has proven 
to be a significant predictor of the odds of 
retention, with full-time students significantly 
more likely to be retained than part-time stu- 
dents. 

Paradoxically, though, online develop- 
mental writing may loom as an obstacle in the 
perspective of many developmental educators, 
even though it has the potential to knock 
down many of the hurdles developmental stu- 
dents often face such as problems with child 
care, transportation, and parking; scheduling 
class times around family and work responsi- 
bilities; feeling part of the academic commu- 
nity; and becoming more proficient with cur- 
rent technology. Furthermore, if online edu- 
cation continues its rapid growth pattern, it 
is possible that developmental students may 
be disadvantaged by limiting online access as 
an option at the beginning of their academic 
work and thus limiting their full preparation 
for access and success in subsequent commu- 
nity college and university work. 
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